Remove ObjectStore::append #5016
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tustvold
commented
Nov 1, 2023
| @@ -94,8 +94,7 @@ | |||
| //! | |||
| //! This provides some compelling advantages: | |||
| //! | |||
| //! * Except where explicitly stated otherwise, operations are atomic, and readers | |||
| //! cannot observe partial and/or failed writes | |||
| //! * All operations are atomic, and readers cannot observe partial and/or failed writes | |||
crepererum
approved these changes
Nov 1, 2023
|
cc @metesynnada |
|
So as I understand t, the idea here that if people want to read from object storage / local files, but append to local files, they need to implement |
|
I would phrase it as, if you have a use-case that involves reading and writing local files, including appending to them, or mmaping them, or some other filesystem specific functionality, then you should use the filesystem directly. apache/datafusion#8021 contains an example of doing this for FIFO files in the context of DataFusion |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #.
Rationale for this change
ObjectStore::append has always been somewhat at odds with the design goals of the crate, as it is not an object store API, and has rather unfortunate characteristics with regards to atomicity. It was largely added to accommodate streaming workloads in DataFusion that are overloading the file IO abstractions.
apache/datafusion#7994 tracks addressing this upstream, and is an effort I intend to drive to completion. Whilst removing it is slightly jumping the gun, I want this to make the next release #5010 which will still give plenty of time to prepare DataFusion for this change
This functionality was only implemented by InMemory and LocalFileSystem and so applications can easily reproduce this functionality should they so desire.
What changes are included in this PR?
Are there any user-facing changes?